Clinical Data Journey (Phase 3): From Metadata Specifications to metacore Objects

Objective

In the previous article, we introduced a metadata-assisted approach by separating SDTM variable definitions from programming logic. While a simple CSV specification is useful for understanding metadata concepts, real-world clinical programming workflows often use structured metadata specifications containing dataset definitions, variable attributes, controlled terminology, and derivations.

In this article, we will introduce the metacore package, convert a structured SDTM specification into a metacore object, and explore how metadata objects provide the foundation for standardized and scalable SDTM programming workflows.

Load Required Packages

library(tidyverse)
library(metacore)

Introduction to Structured Metadata Specification

A CDISC metadata specification is not a single table. It contains multiple linked components.

A typical SDTM specification workbook contains several linked metadata components:

├── Study information

├── Dataset definitions

├── Variable definitions

├── Value-level metadata

├── Controlled terminology

├── Derivations

└── Documentation

Unlike the simple CSV metadata used in Phase 2, a production SDTM specification organizes metadata into multiple related components. Together, these components describe the complete structure of the study.

Import the SDTM Specification

spec_file <- system.file(
  "extdata",
  "SDTM_spec_CDISC_pilot.xlsx",
  package = "metacore"
)

spec_file

[1] "C:/Users/Admin/AppData/Local/R/win-library/4.6/metacore/extdata/SDTM_spec_CDISC_pilot.xlsx"

spec <- read_all_sheets(spec_file)

names(spec)

 [1] "Study"        "Datasets"     "Variables"    "ValueLevel"   "WhereClauses"
 [6] "Codelists"    "Dictionaries" "Methods"      "Comments"     "Documents"   

read_all_sheets() imports the specification workbook into separate metadata tables.This allows each metadata component to be explored individually before combining them into a single metacore object.

Explore Dataset Metadata

head(spec$Datasets)

# A tibble: 6 × 9
  Dataset Description          Class Structure Purpose `Key Variables` Repeating
  <chr>   <chr>                <chr> <chr>     <chr>   <chr>           <chr>    
1 AE      Adverse Events       EVEN… One reco… Tabula… STUDYID,USUBJI… Yes      
2 CM      Concomitant Medicat… INTE… One reco… Tabula… STUDYID,USUBJI… Yes      
3 DM      Demographics         SPEC… One reco… Tabula… STUDYID,USUBJID No       
4 DS      Disposition          EVEN… One reco… Tabula… STUDYID,USUBJI… Yes      
5 EX      Exposure             INTE… One reco… Tabula… STUDYID,USUBJI… Yes      
6 LBCH    Laboratory Tests Re… FIND… One reco… Tabula… STUDYID,USUBJI… Yes      
# ℹ 2 more variables: `Reference Data` <chr>, Comment <chr>

spec$Datasets %>%
  filter(Dataset == "DM")

# A tibble: 1 × 9
  Dataset Description  Class         Structure Purpose `Key Variables` Repeating
  <chr>   <chr>        <chr>         <chr>     <chr>   <chr>           <chr>    
1 DM      Demographics SPECIAL PURP… One reco… Tabula… STUDYID,USUBJID No       
# ℹ 2 more variables: `Reference Data` <chr>, Comment <chr>

The Datasets sheet contains dataset-level metadata such as the dataset name, description, class, purpose, structure, key variables, and whether multiple records per subject are allowed.

Explore DM Variable Metadata

dm_variables <- spec$Variables %>%
  filter(Dataset == "DM")

head(dm_variables)

# A tibble: 6 × 16
  Order Dataset Variable Label    `Data Type` Length `Significant Digits` Format
  <chr> <chr>   <chr>    <chr>    <chr>       <chr>  <chr>                <chr> 
1 1     DM      STUDYID  Study I… text        12     <NA>                 <NA>  
2 2     DM      DOMAIN   Domain … text        2      <NA>                 <NA>  
3 3     DM      USUBJID  Unique … text        11     <NA>                 <NA>  
4 4     DM      SUBJID   Subject… text        4      <NA>                 <NA>  
5 5     DM      RFSTDTC  Subject… date        10     <NA>                 <NA>  
6 6     DM      RFENDTC  Subject… date        10     <NA>                 <NA>  
# ℹ 8 more variables: Mandatory <chr>, Codelist <chr>, Origin <chr>,
#   Pages <chr>, Method <chr>, Predecessor <chr>, Role <chr>, Comment <chr>

The Variables sheet stores metadata for each SDTM variable, including the variable name, label, data type, length, role, origin, and other implementation details.

Compared with the simple DM_Metadata.csv created in Phase 2, this specification contains a much richer description of each SDTM variable, including its role, origin, controlled terminology, and derivation information.

Explore Controlled Terminology

spec$Codelists %>%
  filter(str_detect(tolower(Name), "sex"))

# A tibble: 6 × 8
  ID     Name   `NCI Codelist Code` `Data Type` Order Term  `NCI Term Code`
  <chr>  <chr>  <chr>               <chr>       <chr> <chr> <chr>          
1 SEX    SEX    C66731              text        1     F     C16576         
2 SEX    SEX    C66731              text        2     M     C20197         
3 SEX    SEX    C66731              text        3     U     C17998         
4 SEXPOP SEXPOP <NA>                text        1     BOTH  <NA>           
5 SEXPOP SEXPOP <NA>                text        2     F     <NA>           
6 SEXPOP SEXPOP <NA>                text        3     M     <NA>           
# ℹ 1 more variable: `Decoded Value` <chr>

Metadata stores controlled terminology separately.

For example, the SEX codelist defines the permitted SDTM values:

F = Female M = Male U = Unknown

Storing controlled terminology separately ensures that SDTM variables consistently use CDISC-defined permitted values across studies.

Create metacore Object

The structured specification can now be converted into a metacore object.

meta <- spec_to_metacore(
  spec_file
)

spec_to_metacore() converts the Excel specification into a structured metadata object that can be queried directly by R packages.

Validate

is_metacore(meta)

[1] TRUE

The Excel specification has now been converted into a structured metadata object.

Explore metacore Object

names(meta)

 [1] ".__enclos_env__" "supp"            "codelist"        "derivations"    
 [5] "value_spec"      "var_spec"        "ds_vars"         "ds_spec"        
 [9] "clone"           "metacore_filter" "validate"        "print"          
[13] "initialize"     

The metacore object stores multiple metadata components such as dataset specifications (ds_spec), variable specifications (var_spec), dataset-variable relationships (ds_vars), value-level metadata (value_spec), controlled terminology (codelist), and derivations.These metadata components can be accessed programmatically, allowing SDTM workflows to reference metadata directly instead of hard-coding dataset specifications within R scripts.

SDTM Specification Workbook
            │
            ▼
   read_all_sheets()
            │
            ▼
   Specification Tables
            │
            ▼
   spec_to_metacore()
            │
            ▼
      metacore Object
            │
            ▼
Metadata-driven SDTM Programming

Compare Phase 2 vs Phase 3

Phase 2	Phase 3
Simple CSV metadata	Structured SDTM specification
Single metadata table	Multiple linked metadata tables
Metadata-assisted programming	Metadata object creation
Variable selection controlled in R	Metadata stored in a metacore object

Conclusion

In this phase, we moved from a simple metadata-assisted workflow to a structured metadata-driven approach. We imported a CDISC SDTM specification workbook, explored its metadata components, and converted it into a metacore object. By organizing dataset definitions, variable attributes, controlled terminology, and derivation metadata into a single structured object, metacore provides the foundation for standardized, reusable, and scalable SDTM programming workflows.

In the next phase, we’ll explore how metatools and sdtm.oak can use metacore metadata objects during SDTM dataset development.